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EISH ^HRTTMn MACHINE. 

The present invention relates to an apparatus for sorting flexible 
food products, particularly fish, and a method for operating that 
apparatus. Particularly provided is an apparatus capable of sorting 
fish into species or groups thereof on the basis of shape and colour. 

The provision of apparatus for automatically sorting fish by species, 
size and weight, for example on board fishing or factory vessels, 
makes it possible to automate the collection of data on catch and 
discards. This has important implications for management of fish 
stocks and operation of the common fisheries policy. 

The need for automatic sorting of fish by species has been discussed 
by Tayama et al (Reito (Tokyo) Refrigeration Vol. 57 No . 661 (1982) 
pp. 1146-1150; Shimdate, M. Mitsubishi Denki Hiko (Tokyo). Vol. 

56 Uq, l (1982) pp. W-48. ). Wagner et al (Lebensmittelindustrie 

Vol.30 No*fi (1983) PP. 375-376. ; Lebensmittelindustrie Vol. 3 1 * 
(1987) pp. 20-23. and Strachan et al ( Pattern Recognition Vol. 23 
Nq . 5 (1990) pp. 539-5M .; 'Computer Vision for the fish industry* 
edited by Pau, L.F. and Olafsson, R. ♦ Marcel Decker (1992)). 

These researchers have described the shape of fish by length 
measurements, where length is defined as a horizontal straight line 
from the tip of the head to the base of the tail; and by width 
measurements where width is defined as the distance from the top to 
the bottom of the body measured at a number of equidistant points 
along the length. They found that they could sort 4 species of fish 
with a reliability of 95% and 7 species with an accuracy of 90?. A 
further suitable apparatus for carrying out simple fish sorting using 
aspect ratio and area ratio is described in EP 0331390. 

For a sorting machine to be used on board a fishing vessel, up to 30 
species of fish may have to be sorted with a reliability of 99*. It 
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is therefore necessary to use not only shape but also variables such 

as colour or shading to provide additional sorting information. , 

Colour has been used for image analysis and pattern recognition, for ^ 

example to identify spray paint caps ( Berry, D.T, Pattern 

Recognition Letters Vol. 6 (1987) PP. 69-75) # to detect the colour 

codes on resistors ( Bajon et al. 'Identification of multicoloured 

objects using a vision module* Robot Vision and Sensory Control. 6th 

Int. Conf. Paris, Prance. Vol. 10 (1986) pp. 21-30) and to guide 

a robot to selectively pick up petri dishes (Massen et al Proc. Int. 

Conf. Robot Vision and Sensory Controls (1988). pp. 115-122) ). 

Colour has also been used to segment images into sets of uniform 

colour regions ( Tominaga. Proc. 4th Int. Conf. on Pattern 

Recognition, Cambridge, UK (1988) pp. 163-172; Celenk. 'An adaptive 

machine learning algorithm for colour image analysis and processing 1 

Int. Conf. on Manufacturing Systems and Technology, Cambridge, HA, 

USA (1987) No. 4 pp. 403-412; Klinker et al ; SPIE Applications of 

Artificial Intelligence. 937 (1988) pp. 229-244). and to detect 

edges in images ( Nevatia. 'A colour edge detector* 3rd Int. Joint 

Conf. on Pattern Recognition, Coronado, California, USA (1976) pp. 

829-837). 

The aim of the present invention is to provide an apparatus for 
automated colour based sorting of irregular objects, eg. fish, which 
is capable of being integrated with a shape descriptor system, eg 
apparatus, into a system for sorting fish by species. An improved 
shape descriptor system is also provided whereby the orientation of 
the fish with respect to its front and back ends may be automatically 
asessed and thus all other descriptors related to that. 

In order to facilitate comparison of input descriptors with reference 
descriptors in an automatic apparatus it is necessary to store the ) 
latter in some form which can readily be referred to by automatic, 
particularly electronic, means. Such reference descriptors are thus 
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most conveniently stored as values in a computer memory or logic. 

There are approximately 30 species of demersal and 5 of pelagic fish 
landed in Scotland (Department of Agriculture and Fisheries for 
Scotland 1 Scottish Sea Fisheries Statistical Tables 1988'. Crown 
Copyright (1989) 57 pp. ). A preferred apparatus according to the 
invention sorts pelagic and demersal fish species and treats them 
separately. This is convenient because they live in different parts 
of the water column, demersal living on or near the seabed and pelagic 
mainly in shoals near the surface, and hence they are unlikely to be 
caught together. 

The preferred apparatus of the present invention, and the method for 
operating it, have been shown to be effective in experiments sorting 5 
species of pelagic fish and 18 species of demersal fish; these latter 
18 species representing more than 95% of all demersal fish landed in 
Scotland. The invention offers a reliability of sorting the 18 
species of demersal fish of 100% and of the 5 species of pelagic fish 
of 98%. This presents a significant improvement over all other 
results so far published, these quoting a sorting accuracy of only 
90-95X for 4-7 species of fish. 

The present invention provides an apparatus for indicating the type of 
a fish comprising a means for receiving light from the fish and 
generating an image of it in the form of a set of values therefrom, a 
means for storing the set of values in a computer memory, a means for 
assigning the values to areas of the image, and a means for using 
these to determine and indicate the type of the fish, characterised in 
that the apparatus comprises a means for determining values indicative 
of the colour of all or some of the areas of the fish and that the 
type of fish is indicated by means which compares values indicative of 
colour by area with predetermined such values stored in the memory or 
a processor associated therewith, these being characteristic of a 
particular type of fish, and a means which designates the fish as that 
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type having stored values to which the determined values correspond. 

Most advantageously the indication is used to direct the fish to a 
predetermined reception area associated with the type of fish with 
which it is designated to most closely correspond and the apparatus 
thus is incorporated as part of an apparatus for automatically sorting 
fish. 

Most preferably the orientation of the fish with respect to which end 
is that of the head and which is that of the tail is determined. This 
is conveniently carried out by a novel method which determines the 
width of the image at two points set a distance between each end and 
the mid length of the fish image respectively* particularly about one 
tenth of the total fish image length in from each end, as determined 
in linear unbent form. Using this method the larger width measurement 
is indicative of the head end. 

Similarly, the orientation of a flat fish with respect to whether it 
is top or belly side up may be assessed by interpreting colour as 
indicative of the eye side surface and lack of colour as indicative of 
the reverse side. With round fish such as cod, haddock, whiting and 
saithe the upper half of the fish is always darker than the lower half. 
Both head/tail and top/bottom orientation determination offer 
significant advantage in identifying the fish by the present method. 

More preferably the present invention comprises an apparatus for 
indicating the type of a fish comprising 

(a) a means for storing one or more sets of values indicative of 
predetermined length, widths and areas, and the distribution of 
colour and/or light intensity in these areas, each set characteristic 
of an image of a particular type of fish 

(b) means for receiving the image of a fish, the type of 
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which is to be indicated, in a camera; 

(c) means for receiving the image from the camera in the form of a 
set of values and storing these in a memory means; 

(d) means for determining boundary values of the image of the fish; 

(e) means for determining values representing predetermined lengths, 
widths and/or areas of the image and their ratios, 

(f) means for determining values indicative of colour of predetermined 
areas of the image, 

(g) means for determining which end of the image is representative of 
the front end and which is representative of the back end of the fish 
and/or means for determining which is the upper and which is the lower 
surface of the fish, and comparing the oriented values from (d) (e) 
and (f ) with corresponding values stored in the means (a) as 
characteristic of a particular sort of fish; 

(h) indicating means which indicates the type of fish to be that 
having stored values corresponding with the determined values. 

Most advantageously the indication (h) is used to direct fish to a 
predetermined sorting area associated with their type. 

The means for determining the front and back end preferably uses the 
length and width data derived by (e) to carry out the method described 
above based upon width measurements taken at each of two points, one 
between each end and the mid-length, preferably at about one tenth of 
the distance from each end. The means for determining top and bottom 
surfaces preferably uses the values indicative of colour of 
predetermined areas derived by (f) to carry out the method described 
above whereby the darker or coloured surface is designated the top. 
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Thus in a preferred embodiment the present invention provides a 
sorting apparatus comprising an apparatus as described in either of 
the embodiments above wherein the fish are presented to a camera, eg. 
on a conveyor, and their ratios of predetermined length, widths and/or 
areas, and values indicative of the colour of all or some of the 
areas, as derived from the fish images produced by that camera, are 
used to classify the fish, thus generating a control signal which 
directs the fish to an appropriate destination. Using preferred 
embodiments of the present invention it is possible to indicate the 
type of a fish when that fish may be presented in a variety of forms 
due to its flexibility and deformability, ie. to indicate the type of 
fish and preferably to direct the fish to an appropriate destination. 

A particular form of apparatus of the present invention comprises a 
conveyor belt for carrying one or more rows of fish, preferably in 
rows parallel or perpendicular to its direction of travel, a means for 
illuminating the fish, a colour video camera, a memory having a number 
of storage areas connected to the video camera output, a means for 
storing the camera derived image values of the illuminated fish from 
the one or more rows in a storage area of the memory, a means for 
determining the image size and orientation from stored edge values, a 
means for using the stored edge values to generate shape descriptors 
the values of which are stored and compared with those already in the 
memory or the logic of a computer processor associated therewith, and 
a means for determining the colour of predetermined areas within the 
image periphery and storing and comparing values therefor with those 
stored in the memory or the logic. 

One suitable such means for determining shape descriptors and dividing 
the image up into length, width and area values is described in EP 
0331390, wherein values are stored and transferred between bit planes. 

Typically apparatus of the present invention will comprise a colour 
video camera with red, green and blue signal outputs connected to a 
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computer memory whereby a pattern of pixels corresponding to the image 
to be analysed can be produced. This memory is connected to a 
processor for generating the finite elements of aspect and area ratios 
and the average red, green and blue values of selected parts of the 
image. These elements, or descriptors, are compared with values 
previously generated, eg- from calibration of the apparatus using a 
number of fish of known type, ie. fish of known species, or built 
into the computer logic, and a score generated indicative of whether 
or not the fish is of a given type, eg. species. 

It is preferred to determine the length, width and/or area values of 
the fish by back-lighting it in the image receiving step. This is 
conveniently achieved by use of a light source mounted below the 
fish as it is presented to the means for receiving the image, ie. 
the camera. Where a conveyor belt is used this is conveniently made 
of a material which transmits at least a proportion of light emitted 
from a source placed below it such that back-lighting can be achieved 
thereby providing a distinct silhouette. 

It is preferred to determine the colour values of the fish by 
front-lighting the object in the image-receiving step. This is 
particularly advantageously carried out using diffuse light generated 
by directing light from one or more light sources onto a reflective 
surface from which it passes onto the top of the object. Conveniently 
the reflective surface is the inside a a housing in which the image 
receiving means, eg. video camera, is mounted. Preferably the 
reflective surface is of white colouration. The preferred method of 
operating the apparatus of the invention calibrates the camera 
outputs to normalise them before operation. 

It will be realised by those skilled in the art that signals generated 
by the computer can readily be used to operate a series of deflectors 
on a conveyor belt to deflect a fish to its determined sorting 
destination where all fish of that type will be deflected to. 
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TABLE 1: The groups of Aspect ratio and Area Ratio that are used 

to classify fish into various groups according to the method of the 

invention for use with the apparatus of the invention, including 

the two main groups (Groups 1 and 2) requiring colour analysis 



All Fish 




Fish of 
Afigfict ratio 

Angler fish 
Dab 

Flounder 
Lemon Sole 
Megrim 
Plaice 
Redfish 
Skate 
Sole 
Witch 




Fish of 
Area ratio 

>0.*40 (Group 1 herein) 
Dab 

Flounder 

Lemon Sole 

Megrim 

Plaice 

Redfish 

Sole 

Witch 

n. 20-0. Ho 
Angler fish 



0.20-0. to \ 




(Group 2 herein) \ 




Catfish 


<O.20 


Cod 


Skate 


Coley 




Haddock 




Hake 




Red mullet 




Redfish 




Whiting 


>O.80 


<Q.2Q 


Ling 


Hake — 


<0.80 


Ling 


Hake 
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A method for operating the apparatus of the present invention is also 
provided based upon use of multivariate analysis, particularly 
discriminant analysis and is that disclosed in the Examples described 
below, although other multivariate data processing technoques such as 
neural networking may equally be applied with success. The apparatus 
and method of the present invention will now be illustrated with 
reference the non-limiting examples. Examples 1 to 3 below, further 
embodiments falling within the scope of the claims will occur to those 
skilled in the art in the light of these. Figures 1 to 8 are provided 
to illustrate aspects of the Examples. 



FIGURES 



Figure 1 shows a shape grid constructed for a whiting in two different 
positions, the grid being shown superimposed upon the fish silhouette. 

Figure 2 shows a grid suitable for a flatfish as shown superimposed 
upon the silhouette of a lemon sole (upper image) and a megrim (lower 
image) . 

Figure 3 shows the position of one particular significant coloured 
area (shown as a light area) in the grid for a flounder, a group 1 
fish by the method of Table 1. 

Figure 4 shows the position of some particular significant coloured 
areas (shown as light areas) in the grids of catfish (upper silhouette) 
and cod (lower silhouette), group 2 fish by the scheme of Table 1. 

Figure 5 shows the position of some particular significant coloured 
areas (shown as light areas) in the grids of herring (upper 
silhouette) and sprat (lower silhouette), pelagic fish. 

Figure 6 shows a flow chart setting out the stages in determination of 
the type of fish from their images, backlit and frontlit. as provided 
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by aii embodiment of apparatus of the invention and a method of 
operating it. 

Figure 7 shows a diagrammatic representation of a perspective view of 
a fish sorting apparatus of the present invention. 

Figure 8 shows a diagrammatic representation of the arrangement 
of the means for determining values indicative of colour of areas of 
the object in a preferred embodiment of the invention and its 
relationship to the other parts of the apparatus of Figure 7- 

eyampi.f. i: ftaaaeaiaia o f t»"» imenfciQP ™« "Rthort nf nrrrnt,1ng it. 

A fish sorting apparatus according to the present invention (see 
Figure 7) was provided comprising quartz halogen lamps (4) GEC 300 W 
power with a colour temperature of 2854 K. placed in a housing (2) 
over a partially transparent conveyor belt (1). The interior of the 
housing was matt white in colour such that the light from the lamps 
(J») was reflected from its walls and ceiling to give the diffuse 
lighting that is required to reduce the amount of specular reflection 
from the glossy skin of the fish. 

A video camera (3). Sony DXC 325PK three chip colour video camera, was 
suspended from the top of the housing and before operation of the 
apparatus the automatic gain control was switched off. The images of 
the fish passed before the camera were taken both backlit using a 
lightbox (5) placed between the runs of the conveyor to give the 
silhouette, and frontlit to give the colour. Red. Green and Blue 
values were fed from the camera to a computer arrangement (described 
below and in Figure 8) which was used to determine the various values 
from which descriminant analysis was used to determine fish type 
(species). Outputs from the computer (9) were used to selectively 
operate deflector elements (7) on a diverter conveyor (6) such that 
fish that had been typed were deflected into an appropriate one of 
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a number of species bins (8). 

The colour values were obtained from the red. green and blue (RGB) 
outputs from the video camera. These signals were then digitised to 
eight bits per colour using a frame grabbing board (Imaging Technology 
Inc. or Sprynt Colour Input Board) and then stored on a computer 
workstation (486PC Sun 3"l60C) where they were processed by an i860 
CPU using algorithms written in the C programming language. 

Colour Calibration: The black (R(b) . G(b) and B(b» and white (R(w) . 
Q(w) and B(w) camera output signal levels were obtained by lens 
capping and use of a reference white tile with the camera aperture set 
at ft respectively. The red. green and blue (Rp. Gp. Bp) camera 
outputs were then normalised according to the black and white levels, 
using the following equations: 



» n r - Rfbl x 1002 


(1) 


R(w) - R(b) 




- o p - Glh) x 100? 


(2) 


G(w) - G(b) 




= nr, - Bfb) x 1002 


(3) 


B(w) - B(b) 




done once a day using a 


Macbeth colour chart. 



SinalS SfaflBB teaCCLatnta, Two descriptors were used for crudely 
sorting the fish. The first was the aspect ratio (A,). 
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where w is the maximum width of the fish and 1 is the length of the 
fish from its nose to the end of its tail. 

The second was the area ratio (A r ) 

A, 

A f = _ (5) 
K 

where A f is the area of the front half of the fish, from the head to 
the midpoint of the fish, and A b is the area of the back half of the 
fish, from the tail to the midpoint of the fish, 

Png-fMnn R^f^nre Sy stem and Shape Descriptors: Flatfish are fish 
with large aspect ratios (A s > 0.40) and do not bend very much. But 
fish with small values (A. < 0.40) bend and deform relatively easily 
(Webb and Weihs, 'Fish Biomechanics 1 , Praeger Publishers, New York, 
USA. (1983)). To sort fish by species it is useful to analyse, for 
example, how the width changes along its length and the colouration 
across the fish. To do this a position reference coordinate system 
should be established which can accommodate fish deformations and 
bending. 

Advantageously, the orientation of the fish with respect to which end 
is that of the head and which is that of the tail was determined. 
This was conveniently carried out by determining the width of the 
image at a distance of about one tenth of the total image length in 
from each end. Using this method the larger measurement is indicative 
of the head end. Similarly, the orientation of a flat fish with 
respect whether it was top or belly side up was assessed by 
interpreting colour as indicative of the eye side surface and lack of 
colour as indicative of the reverse side. With round fish such as 
cod, haddock, whiting and saithe the upper half of the fish is always 
darker than the lower half and this information is also usefully used 
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in orienting fish images. 

For each fish image the computer processor was used to construct a 
grid, typically consisting 36 quadrilateral elements as in Fig !• 
Strachan et al (J. Photographic Sci. 1990) have named this grid the 
'shape grid* because it is derived from the shape of the fish. Shown 
is the shape grid for a whiting in two different deformation states 
showing that the grids bend with the fish. 

For each fish type a shape grid was constructed as follows: 

(i) The image of the fish silhouette was thresholded, made binary, 
the principal axis was found and the fish silhouette oriented on it 
horizontally (see Strachan: Pattern Recognition (1990) as above), 

(ii) The width of the fish was determined at each point along its 
length by dropping vertical lines from its top edge to its bottom edge 
and the locus of the midpoints of these lines produced the central 
line of the grid, 

(iii) From the width profile of the fish along its length the tip of 
the head and the narrowest point of the tail stalk were detected, 

(iv) The central line was approximated at 10 segments of equal length 
joining the tip of the head to the narrowest point of the tail stalk 

(v) The normals of the central line were drawn from each of the 
points joining these segments to both the upper and lower edge of 
the fish image, 

(vi) Each normal (both upper and lower) was split into two equal 
segments, and 



(vii) All of these segments were connected to produce the shape grid. 
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It can be seen that such a grid can be constructed to bend with the 
fish and thus is suitably constructed by a computer from values in a 
memory, ie. data in the form of pixels, representative of the image 
of a bent fish. The bent grid may be transformed into a straight grid 
or vice versa by processing with algorithms using method of Books tein. 
Transformations of quadrilaterals, tensor fields, and morphogenesis. 
In P L Antonelli (Ed.) Mathematical Essays on Growth and the Emergence 
of Form. University of Alberta Press, Edmonton, Alberta (1985). 

Such a grid is capable of overlapping itself and hence a different 
sort of grid is required for fish with large aspect ratios, ie. 
flatfish. The grid shown in Fig 2 is suitable for flatfish. It is 
constructed using vertical lines instead of normals to the central 
symmetry line. This grid will be poor at modelling the bending of 
fish but as has been mentioned already flatfish bend very little and 
hence this problem can be ignored • 

From either type of shape grid a set of descriptors can be obtained to 
describe the shape of the fish. These were the 10 widths of the fish 
defined by the grids and the length of the fish from its nose to the 
apex of its tail. These shape descriptors were used to sort the 
fish by species. 

Colour Degrriptors: For the tail and the nose (which in the present 
example consists of the front four shape grid elements) and for each 
of the 36 other elements of the shape grid (or the simple shape grid), 
the average R, G and B values are determined. This produces 114 
variables (36 elements plus nose and tail with average R, G, B in each 
« 38 x 3 « 114) which can be used to sort the fish by species. 

n-tfirriminpnr. Analysis: Discriminant analysis (Nie et al 'Statistical 
package for the social sciences, 2nd ed. , McGraw-Hill, New York, 
U.S.A. (1975). ) is used to process the shape descriptor and the 
colour descriptor data derived from the image in the memory store, 
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employing linear combinations of variables to distinguish between the 
different species of fish. 

The use of discriminant analysis requires the linear combination of 
the variables that best distinguishes between the different species of 
fish to be found and used by the processor that carries out the 
generation of the descriptors from the colour image* A stepwise 
method was used to generate discriminant functions by introducing one 
variable at a time. If the new function satisfied the discriminant 
analysis criterion then it was accepted. If not, it was rejected 
along with the last variable introduced. 

At the end of this process the resulting classification coefficients 
C tJ (Fisher's linear discriminant functions) were obtained. C k , the 
classification score for species group i, is given by a linear 
combination of C ls *8i 

jmax 

C, * £ C w Q(j) ♦ C lo (6) 
j"l 

where i = 1,2 ... n (n * no of species of fish), C lo is a constant 
and the Q(j).'s are the raw values of the variables. 



The variables that were used in Examples 1 to 3 are: 



1. The eleven shape descriptors (jmax = 11) 

2. The R, G, B average values of the nose and front third shape grid 
elements of the fish (jmax = 39) 

3. The R, G, B average values of the middle third shape grid elements 
of the fish (jmax = 36) 
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4. The R, G f B average values of the tail and tall third shape grid 
elements of the fish (jmax = 39) 

The colour descriptors were split into three subgroups because in any 
technique using multivariate analysis the analysis performs better if 
the number of variables is reduced (Digby, P.G.N, and Kempton, R.A. 
'Multivariate Analysis of Ecological Communities • . Chapman and Hall, 
London, England (1987). 

Since there were n species of fish, every new fish to be sorted for 
species was given n scores, one from each classification function. 
Fish were sorted as the species whose function had the highest score. 

For the discriminant analysis the fish have to be split into two sets 
(the calibration and the test set). The species of the calibration 
set fish along with the variables is entered into the processor and 
from these the discriminant analysis algorithm produces the Fisher 
classification coefficients. The variables of the test set of fish 
are then introduced to the processor and this determines the species 
of these fish using the Fisher linear discriminant functions. 

The arrangement of the processor with respect to the R, G, B input 
from the video camera and its associated equipment is shown in Figure 
8. The R, G, B analog inputs from the camera are fed to respective 8 
bit analog to digital converters connected in turn to look up tables 
and 2 megabytes of FIFO store. This store is accessed by a 64 bit 
data bus further connected to a video random access memory and a video 
monitor, an 8 megabyte DRAM and the central processing unit (CPU) . 
The CPU has outputs which control the deflectors of the divertor conveyor. 

In experiments carried out using apparatus configured as described in 
Example 1 and in Figures 5 to 7 there were at most 35 fish of each 
species, hence where possible 20 fish of each species are used as the 
calibration set. This is because the larger the calibration set the 
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better the discriminant analysis performs. 

The aforesaid apparatus, with the processor programmed to carry out 
descriminant analyis as described was provided with fish of either 
Demersal (Example 2) or Pelagic (Example 3) types, these not usually 
requiring sorting together. 

EXAMPLE 2: Apparatus for sorting Desmersal fish. 

The apparatus was set up such that the computer processor first sorted 
images according to aspect ratio A s and then according to their area 
ratio A r (see Table 1 in introduction ). This identified angler fish, 
skate, ling and hake. There remained two groups of fish which 
required further sorting. Group 1 consisted mainly of flat fish and 
some round fish which have large aspect ratios. Group 2 consisted 
exclusively of round fish. 18 fish species were sorted in total. 

For Group 1 the simple shape grid was generated for each of the 
species of fish from which shape and colour descriptors were generated 
(example position of colour varying area shown in Fig 3) and these 
were subjected to discriminant analysis. For the calibration set the 
discriminant analysis sorted all of the fish (135 fish) correctly ie. 
100% sorting reliability for the shape descriptors and the 3 sets of 
colour descriptors. For the test set of fish the shape descriptors 
had a sorting reliability of 9^% (92 out of 98), the front third 
colour descriptors had a sorting reliability of 95% (93 out ot 98) , 
the middle third colour descriptors had a sorting reliability of 95% 
(93 out of 98) and the tail third colour descriptors had a sorting 
reliability of 9**% (92 out of 98). These four sets of results were 
combined by simply classifying a fish as the one most preferred by 
adding the results together. In doing this the sorting reliability 
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obtained was 100% (98 out of 98). 



For the Group 2 fish the shape grid was used. From this the shape and 
colour descriptors were obtained (examples of colour varying areas 
shown in Fig 4) and these were then subjected to discriminant 
analysis. For the calibration set the discriminant analysis sorted 
all of the fish (117 fish) correctly, ie. a 100% sorting reliability 
for the shape descriptors and the 3 sets of colour descriptors. For 
the test set the shape descriptors had a sorting reliability of 90% 
(69 out of 77), the front third colour descriptors had a sorting 
reliability of 96% (74 out of 77). the middle third colour descriptors 
had a sorting reliability of 100% (77 out of 77) and the tail third 
colour descriptors had a sorting reliability of 100% (77 out of 77). 
When these four sets of results were combined as before a sorting 
reliability of 100% (77 out of 77) was obtained. 

Hence the 18 species of demersal fish used in these experiments were 
sorted with a reliability of 100% (133/133 test fish). 

EsaaBlfi v Annflrfltus for sorting Pelagic fish, 

The five main species of pelagic fish have similar values of A r and 
A 8 therefore the procedure of Table 3 does not work on pelagic species. 

Shape grid were generated for all of the pelagic fish and from these 
the shape and colour descriptors were calculated (see examples of 
position of colour varying areas Fig 5) and subjected to discriminant 
analysis. For the calibration set the discriminant analysis sorted . 
all of the fish (80 fish) correctly ie a 100% sorting reliability for 
the shape descriptors and the 3 (front, middle and tail) sets of 
colour descriptors. For the test set of fish the shape descriptors 
had a sorting reliability of 98% (55 out of 56), the front third 
colour descriptors had a sorting reliability of 100% (56 out of 56), 
the middle third colour descriptors had a sorting reliability of 88% 
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(49 out of 56) and the tail third shape descriptors had a sorting 
reliability of 98* (55 out of 56). When these four sets of results 
were combined (as in the previous section) a sorting reliability of 
982 (55 out of 56) was obtained. The one fish which was not 
classified correctly was a sprat and it was classified as 50% 
probability of being sprat and 50* herring. 
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CLAIMS. 

1. An apparatus for indicating the type of a fish comprising a means 
for receiving light from the fish and generating an image of it in the 
form of a set of values therefrom, a means for storing the set of 
values in a computer memory, a means for assigning the values to areas 
of the image of the fish, and a means for using these to determine and 
indicate the type of the fish, characterised in that the apparatus 
comprises a means for determining values indicative of the colour of 
all or some of the areas of the fish and that the type of fish is 
indicated by a means which compares the values indicative of colour by 
area of the fish with predetermined such values stored in the memory 
or a processor associated with it. these being characteristic of a 
particular type of fish, and a means which designates the fish as that 
type having stored values to which the determined values correspond. 

2. An apparatus as claimed in claim 1 wherein the indication is used 
to control a sorting mechanism for directing the fish to a 
predetermined reception area associated with the type of fish with 
which it is designated to most closely correspond. 

3. An apparatus as claimed in claim 1 or claim 2 wherein the 
orientation of the image of the fish is determined with respect to 
which end is the front end of the fish and which is the back end 
and/or which is the upper surface and which is the lower surface, 
and the colour and shape descriptors are related to these. 

H. An apparatus for indicating the type of a fish comprising 

(a) a means for storing one or more sets of values indicative of 
predetermined length, widths and areas, and the distribution of colour 
in these areas, each set characteristic of an image of a particular 
sort of fish; 
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(b) means for receiving the image of a fish, the type of which is to 
be indicated, in a camera; 

(c) means for receiving the image from the camera in the form of a set 
of values and storing these in a computer memory means; 

(d) means for determining boundary values of the image of the fish; 

(e) means for determining values representing predetermined lengths, 
widths and/or areas of the image and their ratios, 

(f) means for determining values indicative of colour of predetermined 
areas of the image, 

(g) means for determining which end of the image is representative of 
the front end and which is representative of the back end of the fish 
and/or means for determining which is the upper and which is the lower 
surface, and comparing the oriented values from (d) (e) and (f) with 
corresponding values stored in the means (a) as characteristic of a 
particular sort of fish; 

(h) indicating means which indicates the type of fish to be that 
having stored values corresponding with the determined values. 

5. An apparatus as claimed in claim H wherein the indication means 
controls a mechanisms which directs the fish to a predetermined 
sorting reception area associated with the type of fish. 

6. A sorting apparatus as claimed in any one of claims 1 to 4 wherein 
the fish are presented to a camera and their ratios of predetermined 
length, widths and/or areas, and the values indicative of colour in 
all or some of the areas, as derived from the image produced by that 
camera, are used to classify the fish, thus generating a control 
signal which directs the fish to an appropriate destination. 



PCT/GB93/021S1 

WO 94/09920 



22 

7. An apparatus comprising a conveyor belt for carrying one or more 
rows of fish, a means for illuminating the fish, a colour video 
camera, a video memory having a number of storage areas connected to 
the video camera, a means for storing the camera derived image values 
of a fish from the one or more rows in a first storage area of the 
memory, a means for determining the image size and orientation from 
stored edge and/or colour values, a means for using the stored edge 
values to generate shape descriptors the values of which are stored 
and compared with those already in the memory or logic of a computer 
processor associated therewith, and a means for determining the colour 
and/or light intensity of predetermined areas within the image 
periphery and storing and comparing values therefor with those stored 
in the memory or logic. 

8. An apparatus as claimed in any one of the preceding claims 
comprising a colour video camera with red. green and blue signal 
outputs connected to a computer memory whereby a pattern of pixels 
corresponding to the image to be analysed can be produced, a processor 
for generating the finite elements of aspect and area ratios and the 
average red. green and blue values of selected parts of the image, the 
processor being connected to the memory, a comparator for comparing 
these elements with values previously generated from calibration of 
the apparatus using a number of fish of known type and stored in 

the memory or incorporating in the logic of a processor associated 
therewith, and a means for generating a score indicative of the number 
whether or not the fish is of a given type. 

9. An apparatus as claimed in any one of the preceding claims 
wherein the image of the fish is recorded by the camera under both 
backlit and frontlit lighting conditions. 

10. An apparatus as claimed in any one of the preceding claims 
wherein signals from the red. green and blue (RGB) outputs from the 
video camera are digitised to eight bits per colour using a frame 
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grabbing board and then stored on a computer workstation where they 
are processed using algorithms. 

11. An apparatus as claimed in any one of the preceding claims 
wherein the orientation of the fish with respect to which end is the 
head and which end is the tail is carried out by determining the width 
of the image at each of two points, each placed between a 
respective end and the mid-length of the image, and designating the 
larger width to be indicative of the head. 

12. An apparatus as claimed in claim 11 wherein the two points are 
situated at a distance approximately one tenth of the total image 
length from each end. 

13. An apparatus as claimed in any one of the preceding claims 
wherein the orientation of the fish with respect to which surface is 
the upper surface and which surface the lower surface is carried out 
by designating the lighter or less coloured surface the lower surface. 

14. An apparatus as claimed in any one of the preceding claims 
wherein the image values are used to generate aspect ratio (AJ and 
area ratio (A r ) shape descriptors. 

15. An apparatus as claimed in claim 14 wherein those images having 
aspect ratios of over 0.40 are indicated as those of flatfish and 
those below 0.40 as roundfish. 

16. An apparatus as claimed in claim 15 wherein the image of the 
fish has an aspect ratio of over 0.40 and is used to generate a grid 
constructed using vertical lines from the central line of symmetry. 

17. An apparatus as claimed in claim 14 wherein the image of the fish 
has an aspect ratio less than 0.40 and is used to generate a grid of 
quadrilateral elements constructed from lines normal to its central 
line of symmetry. 
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18. An apparatus as claimed in either of claims 16 or 17 wherein 
the grid has a number of width values generated from the fishes nose 
to its tail. 

19. An apparatus as claimed in any one of the preceding claims 
wherein the average R, G f B values of the image are determined for the 
tail, the nose and for some or all of the elements of the shape grid, 
having a computer processor which carries out multivariate analysis 
processing using these values as descriptor values for determining the 
species of the fish. 

20. An apparatus as claimed in claim 19 wherein the width values are 
used as descriptor values for multivariate analysis processing 
wherein they are used to determine the species of the fish. 

21. An apparatus as claimed in claim 19 or 20 wherein the linear 
combination of the variables that best distinguishes between the 
different species of fish to be sorted is installed in the logic of 
the processor that carries out the multivariate analysis. 

22. An apparatus according to any one of claims 19 to 21 wherein the 
multivariate analysis is discriminant analysis. 

23. An apparatus as claimed in any one of claims 19 to 22 wherein 
discriminant functions are generated by introducing one variable at a 
time wherein if the new function satisfies discriminant analysis 
criterion then it is accepted and if not, it is rejected along with 
the last variable introduced. 

2k. An apparatus as claimed in claim 23 wherein the classification 
score for species group i, is given by a linear combination of C^'s: 



jmax 

= £ C,j Q(j) ♦ C 1o 



(6) 
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where i = 1,2 ... n (n = no of species of fish), C io is a constant 
and the Q(j)'s are the raw values of the variables. 

2k. An apparatus as claimed in any one of claims 21 to 23 wherein the 
variables used comprise width shape descriptors, a length descriptor, 
the R, G, B average values of the nose and front third shape grid 
elements of the fish, the R, G, B average values of the middle third 
shape grid elements of the fish, and the R, G, B average values of 
the tail and tail third shape grid elements of the fish. 

25. An apparatus as claimed in any one of claims 21 to 2k wherein 
fish to be sorted for species are given n scores, one from each 
classification function, and are sorted as the species whose function 
had the highest score. 

26. An apparatus as claimed in claim 25 adapted such that when the 
species of a calibration set of fish inputted to the processor along 
with the variables, the discriminant analysis algorithm produces the 
Fisher classification coefficients, such that when the variables of 
the test set of fish are introduced to the processor it can determine 
the species of these fish using the Fisher linear discriminant 
functions . 

27. An apparatus as claimed in any one of the preceding claims 
suitable for indicating type of and/or sorting desmersal fish 
characterised in that it first sorts the fish images according to 
aspect ratio (AJ, then according to area ratio (A r ), as defined 
herein, then subject to multivariate analysis. 

28. An apparatus as claimed in any one of claims 1 to 27 
substantially as described in any one of the Examples 1, 2 or 3. 

29. A method for operating an apparatus as claimed in any one of the 
claims 1 to 27 substantially as described herein in any one 

of Examples 1, 2 or 3- 
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